Method and apparatus for organizing digital images with embedded metadata

ABSTRACT

One embodiment of the present invention provides a system that facilitates automatically organizing digital images using metadata which is embedded in the digital images. The system operates by receiving a digital image with embedded metadata, wherein the embedded metadata includes Global Positioning System (GPS) data. The system uses the embedded metadata, including the GPS data, to identify additional metadata associated with the digital image. The system then saves the digital image on a storage medium. In addition, the system saves the embedded metadata and the additional metadata on the storage medium in a searchable format.

BACKGROUND

1. Field of the Invention

The present invention relates to computer systems that manipulatedigital images. More specifically, the present invention relates to amethod and an apparatus for automatically organizing digital imagesusing metadata which is embedded in the digital images.

2. Related Art

Digital imaging technology is rapidly improving, and the cost of digitalcameras is continuing to decrease. Furthermore, digital cameras are nowbeing embedded in many different devices, such as cell phones, PersonalDigital Assistants (PDAs), and portable MP3 players.

In addition to the improvements in digital imaging technology, onlinephoto sharing networks and online galleries are more prevalent than everbefore. With these modern technologies, it is possible to take hundredsof pictures, view them, touch them up, and annotate them before makingthem available on a public server.

A significant problem with non-digital photography is not being able tosee what the images look like until after the film is developed. Withdigital imaging, you can typically view the image that has been capturedimmediately after the photograph is taken. In addition, memory cards letusers take many more pictures than traditional film based cameras, andeach additional picture incurs little or no additional cost.

The biggest burden with digital photography is to organize and annotatethe digital images. A photographer typically uploads the images and hasto review each one to identify attributes, such as locations, events,and people or objects associated with the images.

Hence, what is needed is a method and an apparatus for organizingdigital images without the problems described above.

SUMMARY

One embodiment of the present invention provides a system thatfacilitates automatically organizing digital images using metadata whichis embedded in the digital images. The system operates by receiving adigital image with embedded metadata, wherein the embedded metadataincludes Global Positioning System (GPS) data. The system uses theembedded metadata, including the GPS data, to identify additionalmetadata associated with the digital image. The system then saves thedigital image on a storage medium. In addition, the system saves theembedded metadata and the additional metadata on the storage medium in asearchable format.

In a variation of this embodiment, the additional metadata includes anidentifier for a location associated with the GPS data.

In a variation of this embodiment, the metadata includes a time at whichthe digital image was acquired, and the additional metadata includes aweather condition for a location associated with the GPS data at thetime the digital image was acquired.

In a variation of this embodiment, using the embedded metadata toidentify additional metadata involves querying a web service.

In a variation of this embodiment, using the embedded metadata toidentify additional metadata involves performing a lookup in a PersonalInformation Manager (PIM) application.

In a further variation, the system queries a user's calendar informationto identify additional metadata including an event associated with thedigital image.

In a further variation, the system queries a user's contact informationto identify additional metadata including a location associated with thedigital image.

In a variation of this embodiment, using the embedded metadata toidentify additional metadata involves performing a lookup in a customdictionary.

In a variation of this embodiment, the system annotates the image withthe additional metadata so that the additional metadata can be displayedto a user.

In a variation of this embodiment, the system uses the metadata toidentify digital images that are similar to the digital image. Oncesimilar digital images are identified, the system presents a user withannotations from the similar digital images as possible options tofacilitate annotating the digital image.

In a further variation, presenting the user with annotations from thedigital images involves presenting the user with annotations that havenot been automatically generated.

In a variation of this embodiment, the system additionally facilitatesbrowsing a collection of digital images organized by metadata.

In a variation of this embodiment, the system adds the digital image toa gallery of images that have metadata similar to the metadata from thedigital image.

In a variation of this embodiment, the system uses the metadata toperform a dynamic lookup query to present a user with a collection ofdigital images.

In a further variation, performing the dynamic lookup query involvesperforming a join operation with the metadata.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates digital imaging in accordance with an embodiment ofthe present invention.

FIG. 2 illustrates a photo database with metadata in accordance with anembodiment of the present invention.

FIG. 3 illustrates the process of storing digital images in accordancewith an embodiment of the present invention.

TABLE 1 illustrates various possible browsing options in accordance withan embodiment of the present invention.

TABLE 2 illustrates various possible search queries in accordance withan embodiment of the present invention.

DETAILED DESCRIPTION

The following description is presented to enable any person skilled inthe art to make and use the invention, and is provided in the context ofa particular application and its requirements. Various modifications tothe disclosed embodiments will be readily apparent to those skilled inthe art, and the general principles defined herein may be applied toother embodiments and applications without departing from the spirit andscope of the present invention. Thus, the present invention is notintended to be limited to the embodiments shown, but is to be accordedthe widest scope consistent with the principles and features disclosedherein.

The data structures and code described in this detailed description aretypically stored on a computer readable storage medium, which may be anydevice or medium that can store code and/or data for use by a computersystem. This includes, but is not limited to, magnetic and opticalstorage devices such as disk drives, magnetic tape, CDs (compact discs)and DVDs (digital versatile discs or digital video discs), and computerinstruction signals embodied in a transmission medium (with or without acarrier wave upon which the signals are modulated). For example, thetransmission medium may include a communications network, such as theInternet.

Background

FIG. 1 illustrates the digital imaging process in accordance with anembodiment of the present invention. During this process, digitalimages, such as digital image 108, are typically associated withmetadata, such as the date the image was taken, the time the image wastaken, the exposure settings, etc. For example, the metadata can beExchangeable Image File Format (EXIF) data which is stored in JointPhotographic Experts Group (JPEG) images.

The addition of digital imaging technology and Global Positioning System(GPS) capabilities to cell phones 106 and Personal Digital Assistants(PDAs) 104 has enabled some additional metadata to be captured when thedigital image 108 is taken. One embodiment of the present invention usesthis GPS location information, along with other metadata 110 imbedded inthe digital image 108 to organize, share, browse and search a collectionof digital photographs on behalf of a user 100 or a group of users.

Currently, when an image is captured by a digital camera 102, thedigital camera 102 typically stores metadata 110 in the image header,such as date the image was captured, time of the day the image wascaptured, camera make and model, exposure settings, ISO settings, etc.In addition to this metadata 110, in one embodiment of the presentinvention, location-specific information, such as the coordinates of theplace where the photograph was taken and the altitude from which theimage was taken is also stored in the embedded image metadata 110. Notethat this is possible with current technology since it simply requires aGPS device to be embedded in a digital camera 102. Furthermore, cellularphones of today are equipped with both digital cameras and GPStechnologies, and they are increasingly used as a primary camera devicefor many users. The newest generation of cellular phones has betterimaging technology than ever before, and consequently, the number ofusers who use their phone as their primary camera continues to grow.

Organization

In one embodiment of the present invention, embedded metadata 110 can beused to auto annotate digital image 108 and organize digital image 108with other digital images in a collection or albums. Note that in oneembodiment of the present invention, digital images which are storedinside a database, such as an Oracle™ database, can be organized intocollections through the use of database VIEWS over the image data.

The title or annotation for an individual image, such as digital image108, can be generated by obtaining the name of the place where thephotograph was taken either through a dictionary or by connecting to aservice that maps coordinates to place names. For example, a servicecould map coordinates to terms such as “my office”, “my backyard”, orthe Shoreline Amphitheater. Coupled with this time and date information,the system could generate a collection of images that have related timesand dates, such as spring 2003 in Niagara Falls or winter 2002 inAlaska.

Yet another example provides a collection of all Alaska photos, or allphotographs in 2003. In this example, the embedded date information isused to generate season/month and the location information.

Furthermore, the system can tag the photographs with information such asthe weather at the time and location that the photograph was taken by.This can be accomplished by connecting to a weather service andobtaining the conditions at a particular time and place. Note that thepresent invention is not meant to be limited to the examples listedabove. Virtually any combination of data from metadata 110 can be usedto associate digital image 108 with other metadata that would be usefulin annotating and organizing digital image 108.

Browsing

Digital images with embedded metadata coupled with the powerful indexingand integration features of a database can be used to provide amulti-dimensional access to the digital images. A system that organizesimages into a simple file systems allows folder-based viewing of theimages. In contrast, using a database to store images allows flexiblemenu-based viewing. Possible examples of menus are: TABLE 1 All Images-> Year -> 2004 -> Months -> October; All Images -> Year -> 2004 ->Seasons -> Fall; All Images -> Location -> North America -> USA ->Niagara Falls; All Images -> Time of the Day -> Sunset; All Images ->Occasions -> Birthday (Note that this would require integration with apersonal Information database); All Aerial Images -> Location -> NorthAmerica -> USA -> Death Valley.Searching

The metadata stored with the images, along with the annotationsgenerated while organizing the images, can be used to search over theimage collections on behalf of a given user or a group of users. Thesesearches can be based on keywords, or can be performed through astructured column based form, or a combination of the two. Thesesearches can also be based on information from other sources, such asweather information, location information, occasion information, etc.Examples of some queries are: TABLE 2 Niagara Falls; All photographswithin x miles of Eiffel Tower that have count greater than y (Can beused to find preferred tourist spots); Fall photographs on highway 108in Stowe, Vermont; Sunset photographs on highway 1; Britney Spears atBali; All photographs taken in 2001 (from personal collection);Photographs taken in rain last year (links weather info to date).

Searching can also be performed on user-provided annotations that arestored along with the images in an object-relational database. Forexample, a query can find all photographs of Scooby Doo, if thephotographs that contain the image of Scooby Doo are annotated with tag“Scooby Doo”.

Sharing and Collaboration

Embedding location metadata into images facilitates a natural linkbetween photographs taken by different users, which enables users toshare annotations photographs taken at same location. These annotationscan be additionally ranked by the number of other image attributes thatmatch, such as year, time of day, weather conditions, etc. Users canenforce an additional constraint to obtain only those annotations thatwere not automatically generated. The system can then rank annotatorsbased on number of people who select their annotations. This canincrease the quality of the automatic suggestions that the systemprovides. Note that in one embodiment of the present invention, users donot need to upload their images to the service if they do not want toshare the image. However, they can still share the annotations by simplyuploading the image metadata.

Server Technology

In one embodiment of the present invention, the system can begenerically implemented as a feature of an Object Relational Database.Metadata can be automatically populated for images stored in thedatabase to facilitate subsequent retrieval, updates, and miscellaneousapplication support. This plus user-provided metadata can be used toprovide virtual collections of objects in the database.

In another embodiment of the present invention, the database is modifiedto recognize and store implicit associations between objects throughvirtual joins based on metadata collected from various data sources,such as weather databases, GPS databases, and personal informationdatabase. In addition, the database can export a protocol through whichpublic content (which is possibly indexed by a hosting service forcollaboration and sharing) can be exposed. (This is similar to exposingtext content to search engines).

Hosting Technology

In one embodiment of the present invention, the image search capabilitycan be a part of an enterprise search application, as well as astand-alone application. As part of an enterprise search application,the image search capability will provide users the capability to searchthrough image files on the basis of typed in keywords. Each searchresult (image) can be appended with the list of virtual folders (albums)to which this image belongs. This allows the user to navigate to thosealbums to look for related images.

As a stand-alone application, the image search capability can providesearch capability based on keywords similar to the enterprise searchapplication discussed above. However, in this application, the user canalso obtain an album view of the entire image collection. The user willthen be able to modify the albums and the image metadata. Note that theimage collection which is available to the user depends on the currentcredentials of the user, as well as the credentials associated with eachimage/album. For example if a user X is logged into the application anduser Y has specified that,

-   -   All the images in my public album are available to everyone,    -   All the images in my album G-album are available to members of        group G (of which X is a member),    -   All the images in my album X-album are available to X.        Then, user X can view these three albums and the images        contained in them. User Y can also specify which images/albums        are view only and which can be modified by others. This stand        alone application also provides the capability to be able to        upload an image and to add keywords to an image, which will help        facilitate subsequent searches.        Photo Database with Embedded Metadata

FIG. 2 illustrates a photo database 200 with metadata 110 in accordancewith an embodiment of the present invention. Database 200 can includeany type of system for storing data in non-volatile storage. Thisincludes, but is not limited to, systems based upon magnetic, optical,and magneto-optical storage devices, as well as storage devices based onflash memory and/or battery-backed up memory. Database 200 includesimages table 202 and attributes table 204. As images, such as digitalimage 108 are stored in images table 202, the associated metadata, suchas metadata 110, is stored in attributes table 204. This facilitatesindexing and fast retrieval of images from images table 202 via theassociated metadata in attributes table 204.

Process of Storing Digital Images

FIG. 3 illustrates the process of storing digital images in accordancewith an embodiment of the present invention. The system starts when user100 stores digital image 108 into images table 202 in database 200 (step302). The system then retrieves embedded metadata 110 from digital image108 (step 304) and stores metadata 110 in attributes table 204 (step306).

The system also uses the metadata 110 to identify additional metadataassociated with digital image 108 (step 308). Note that this couldinclude connecting to a web service, querying a database, scanning aflat file, examining a PIM application, etc. For example, the systemcould lookup the name of a location associated with GPS coordinates inmetadata 110. Finally, the system stores the additional metadata intodatabase 200 (step 310). Note that it is not necessary to store theadditional metadata in database 200, and the additional metadata can beidentified at a later time. However, it may be necessary to store theadditional metadata into database 200 to facilitate indexing and fasterqueries against the data a later time.

The foregoing descriptions of embodiments of the present invention havebeen presented for purposes of illustration and description only. Theyare not intended to be exhaustive or to limit the present invention tothe forms disclosed. Accordingly, many modifications and variations willbe apparent to practitioners skilled in the art. Additionally, the abovedisclosure is not intended to limit the present invention. The scope ofthe present invention is defined by the appended claims.

1. A method for automatically organizing digital images using metadatawhich is embedded in the digital images, the method comprising:receiving a digital image with embedded metadata, wherein the embeddedmetadata includes Global Positioning System (GPS) data; using theembedded metadata, including GPS data, to identify additional metadataassociated with the digital image; saving the digital image on a storagemedium; and saving the embedded metadata on the storage medium in aformat that can be used by text-based search engines.
 2. The method ofclaim 1, wherein the additional metadata includes a name for a locationassociated with the GPS data.
 3. The method of claim 1, wherein themetadata includes a time at which the digital image was acquired, andwherein the additional metadata includes a weather condition for alocation associated with the GPS data at the specified time.
 4. Themethod of claim 1, wherein using the embedded metadata to identifyadditional metadata further involves querying a web service.
 5. Themethod of claim 1, wherein using the embedded metadata to identifyadditional metadata further involves performing a lookup in a PersonalInformation Manager (PIM) application.
 6. The method of claim 5, furthercomprising querying a user's calendar information to identify additionalmetadata including an event associated with the digital image.
 7. Themethod of claim 5, further comprising querying a user's contactinformation to identify additional metadata including a locationassociated with the digital image.
 8. The method of claim 1, whereinusing the embedded metadata to identify additional metadata furtherinvolves performing a lookup in a custom dictionary.
 9. The method ofclaim 1, further comprising annotating the image with the additionalmetadata for display to a user.
 10. The method of claim 1, furthercomprising: using the metadata to identify digital images that aresimilar to the digital image; and presenting a user with annotationsfrom the digital images to facilitate annotating the digital image. 11.The method of claim 10, wherein presenting the user with annotationsfrom the digital images further involves presenting the user withannotations that have not been automatically generated.
 12. The methodof claim 1, further comprising browsing a collection of digital imagesorganized by metadata.
 13. The method of claim 1, further comprisingadding the digital image to a gallery of images that have metadatasimilar to the metadata associated with the digital image.
 14. Themethod of claim 1, further comprising using the metadata to perform adynamic lookup query to present a user with a collection of digitalimages.
 15. The method of claim 14, wherein performing the dynamiclookup query involves performing a join operation with the metadata. 16.A computer-readable storage medium storing instructions that whenexecuted by a computer cause the computer to perform a method forautomatically organizing digital images using metadata which is embeddedin the digital images, the method comprising: receiving a digital imagewith embedded metadata, wherein the embedded metadata includes GlobalPositioning System (GPS) data; using the embedded metadata, includingGPS data, to identify additional metadata associated with the digitalimage; saving the digital image on a storage medium; and saving theembedded metadata on the storage medium in a format that can be used bytext-based search engines.
 17. The computer-readable storage medium ofclaim 16, wherein the additional metadata includes a name for a locationassociated with the GPS data.
 18. The computer-readable storage mediumof claim 16, wherein the metadata includes a time at which the digitalimage was acquired, and wherein the additional metadata includes aweather condition for a location associated with the GPS data at thespecified time.
 19. The computer-readable storage medium of claim 16,wherein using the embedded metadata to identify additional metadatafurther involves querying a web service.
 20. The computer-readablestorage medium of claim 16, wherein using the embedded metadata toidentify additional metadata further involves performing a lookup in aPersonal Information Manager (PIM) application.
 21. Thecomputer-readable storage medium of claim 20, wherein the method furthercomprises querying a user's calendar information to identify additionalmetadata including an event associated with the digital image.
 22. Thecomputer-readable storage medium of claim 20, wherein the method furthercomprises querying a user's contact information to identify additionalmetadata including a location associated with the digital image.
 23. Thecomputer-readable storage medium of claim 16, wherein using the embeddedmetadata to identify additional metadata further involves performing alookup in a custom dictionary.
 24. The computer-readable storage mediumof claim 16, wherein the method further comprises annotating the imagewith the additional metadata for display to a user.
 25. Thecomputer-readable storage medium of claim 16, wherein the method furthercomprises: using the metadata to identify digital images that aresimilar to the digital image; and presenting a user with annotationsfrom the digital images to facilitate annotating the digital image. 26.The computer-readable storage medium of claim 25, wherein presenting theuser with annotations from the digital images further involvespresenting the user with annotations that have not been automaticallygenerated.
 27. The computer-readable storage medium of claim 16, whereinthe method further comprises browsing a collection of digital imagesorganized by metadata.
 28. The computer-readable storage medium of claim16, wherein the method further comprises adding the digital image to agallery of images that have metadata similar to the metadata associatedwith the digital image.
 29. The computer-readable storage medium ofclaim 16, wherein the method further comprises using the metadata toperform a dynamic lookup query to present a user with a collection ofdigital images.
 30. An apparatus that is configured to automaticallyorganize digital images using metadata which is embedded in the digitalimages, comprising: a receiving mechanism configured to receive adigital image with embedded metadata, wherein the embedded metadataincludes Global Positioning System (GPS) data; an identificationmechanism configured to use the embedded metadata, including GPS data,to identify additional metadata associated with the digital image; astorage mechanism configured to save the digital image on a storagemedium; and wherein the storage mechanism is further configured to storethe embedded metadata on the storage medium in a format that can be usedby text-based search engines.